Harvard Extension School

CSCI E-104: Advanced Deep Learning

Spring 2024

Instructors: Zoran B. Djordjevic & Blagoje Z. Djordjevic

Student: Artemio Mendoza-García

---

Final Project

Vector Databases: Unveiling the Celebrities Among Us

---

Motivation

It is a stablished fact that aliens walk among us, as demonstrated by the documental MIB (Men In Black, Will Smith et all. 1997). However, little we know if a celebrity is disguised as a regular person, living in our same neighborour, goes to the same grocery store than we go, or, if she is taking the same course, CS104-Advanced deep learning, and we are not aware?

We will use a Vector Database, with more than two hundred thousand vector embeddings of more than ten thousand celebrities, to try to shed some light on this potential problematic situation.

But, from where do we get Celebrities images?

We start with the Dataset CELEBA, a well known collection of images from the university of Hong Kong. It contains more than 200,000 images from more than 10,000 celebrities.

For this project, we use the "in-the-wild" images, which are availble for download as a compressed file in this google drive.

The compressed file size exceeds the 1.3 GB mark, and took about one hour to decompress. It was stored locally to speed the insert process.

And, how do we extract the Vector Embeddings?

We use DeepFace, which abstract embedding extraction using different CNN SOTA models, among them:

• FaceNet (128d and 512d)
• VGG_Face
• OpenFace
• DeepFace

For this project, we used FaceNet. FaceNet was developed by Google in 2015. With 140 million of parameters and 22-layer depth, it achieves a prediction accuracy of 99.22% on LFW dataset. With this model Google introduced the triplet loss function, which works using forming triplets, with one anchor, positive example, and negative example.

The image above is a high level schema of the FaceNet architecture

What about where tostore the Vector Embeddings?

For this project we are used Milvus, an Open-source Vector Database

Milvus can be installed locally locally or in the cloud. For this project, we installed the Milvus locally, as a standalone instance with GPU support, using the Docker image provided in the documentation manual.

The standalone instance includes three components:

• Milvus: the core functional component.
• Meta Store: the medata engine, which access and stores metadata of Milvus'internal components, including proxies, index nodes, and more.
• Object Storage: The storage engine, which is responsible for data persistence for Milvus

Ok, we have all we need, now, let's code!

import os
from tqdm import tqdm

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib.gridspec as gridspec
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from scipy.spatial.distance import pdist, squareform
from mpl_toolkits.mplot3d import Axes3D

import cv2
import numpy as np
from IPython.display import display, Image
import pandas as pd
import ipywidgets as widgets
import threading
from datetime import datetime 
from deepface import DeepFace

# Paths for images
CAPTURED_IMAGE_PATH='images'

# Local path with the dataset CELEBA
CELEBA_IMAGE_PATH="D:/Users/artex/Downloads/Compressed/img_align_celeba/img_align_celeba/"

Task 1: Create Database with Vector Embeddings

---

We use Milvus and CelebA

Step 1: Create Milvus Collection

Connect to Milvus

from pymilvus import (
    connections,
    utility,
    FieldSchema,
    CollectionSchema,
    DataType,
    Collection,
)

import pymilvus

# Connect to Milvus
connections.connect(host='localhost', port='19530')

# Get Milvus client version
print("Milvus client version:", pymilvus.__version__)

# Get Milvus server version
server_version = utility.get_server_version()
print("Milvus server version:", server_version)

Milvus client version: 2.4.1
Milvus server version: v2.4.0

Create the collection schema

from pymilvus import FieldSchema, CollectionSchema, DataType

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128),
    FieldSchema(name="x", dtype=DataType.INT32),
    FieldSchema(name="y", dtype=DataType.INT32),
    FieldSchema(name="w", dtype=DataType.INT32),
    FieldSchema(name="h", dtype=DataType.INT32),
    FieldSchema(name="confidence", dtype=DataType.FLOAT),
    FieldSchema(name="name", dtype=DataType.VARCHAR, max_length=255)
]

schema = CollectionSchema(fields, description="Face data collection")

# Create a collection
collection_name = "face_embeddings"
collection = Collection(name=collection_name, schema=schema)

Create the collection Index

# Create an index
index_params = {
    "index_type": "IVF_FLAT",
    "metric_type": "L2",
    "params": {"nlist": 100}
}
collection.create_index(field_name="embedding", index_params=index_params)

Status(code=0, message=)

Verify that the collection was created and its index

collection_list = utility.list_collections()
print("Collections:")
for collection_name in collection_list:
    print(collection_name)

print(f"number of entities: {collection.num_entities}")
print(collection.index)

Collections:
face_embeddings
number of entities: 187647
<bound method Collection.index of <Collection>:
-------------
<name>: face_embeddings
<description>: Face data collection
<schema>: {'auto_id': True, 'description': 'Face data collection', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'embedding', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'x', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'y', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'w', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'h', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'confidence', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'name', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 255}}], 'enable_dynamic_field': False}
>

Step 2: Extract Embedings and Load Milvus DB

Extract embedings using deepface

def get_embeddings(image_path, model_name):
    #extract the face image and embeddings
    # face_image = DeepFace.extract_faces(image_path, enforce_detection=True)

    # Enforce_Detection = True if the image wasn't pre-extracted (extract_features not used before)
    # face_vector = DeepFace.represent(face_image[0]['face'], model_name=model_name, enforce_detection=False)
    face_vector = DeepFace.represent(image_path, model_name=model_name, enforce_detection=True)

            
    # split vector into embeddings, metadata, and confidence
    embedding = face_vector[0]['embedding']
    metadata = face_vector[0]['facial_area']
    confidence = face_vector[0]['face_confidence']   

    return embedding, metadata, confidence

Format Vector Embeddings and Metadata

def fill_data_collection(embedding, metadata, confidence, filename):
    # Fill Data Collection 
    data = [
        {
            "embedding": embedding, 
            "x": metadata['x'],
            "y": metadata['y'],
            "w": metadata['w'],
            "h": metadata['h'],
            "confidence": confidence,
            "name": filename
        },
        # More data entries
    ]

    # Prepare the data for insertion
    embeddings = [entry["embedding"] for entry in data]
    xs = [entry["x"] for entry in data]
    ys = [entry["y"] for entry in data]
    ws = [entry["w"] for entry in data]
    hs = [entry["h"] for entry in data]
    confidences = [entry["confidence"] for entry in data]
    names = [entry["name"] for entry in data]   

    return [embeddings, xs, ys, ws, hs, confidences, names]

Insert Vector embeddings and metadata into Milvus

def insert_image(image_path):
    embedding, metadata, confidence = get_embeddings(image_path, model_name[0])
    if confidence == 0.0: 
        raise ValueError(f"confidence is zero")
    # Format data collection 
    data_collection = fill_data_collection(embedding, metadata, confidence, os.path.basename(image_path))
    # Insert data into Milvus
    insert_result = collection.insert(data_collection)

Function to populate the Vector Database

def populate_vdb(image_directory, idx_start, idx_end):
    # Get list of files
    file_list = [f for f in os.listdir(image_directory)[idx_start:idx_end] if f.endswith(".jpg")]

    # initialize progress bar
    for filename in tqdm(file_list, desc="Processing Images"):
        # Sanity check for type of image
        image_path = os.path.join(image_directory, filename)
        # print(image_path)
        try:
            # Retrives vector embeddings
            insert_image(image_path)
            # embedding, metadata, confidence = get_embeddings(image_path, model_name[0])
            # if confidence == 0.0: 
            #     raise ValueError(f"confidence is zero")
            # # Format data collection 
            # data_collection = fill_data_collection(embedding, metadata, confidence, filename)
            # # Insert data into Milvus
            # insert_result = collection.insert(data_collection)
    
        except Exception as e:
            log_error_images.append({
                'image_path':image_path,
                'error_msg':f"Failed to process {filename}: {str(e)}"
            })   
            # print(log_error_images[-1]['error_msg'])

Now we can populate the collection!

image_directory=CELEBA_IMAGE_PATH
# image_directory='CelebA Dataset/' 
# Get Vector Embeddings and Metadata
model_name = ["Facenet","VGG-Face","OpenFace","DeepFace"]
log_error_images=[]

# Insert only a chunk of Images
img_start_idx = 2000
img_idx_size  = 1500
img_end_idx = img_start_idx+img_idx_size

populate_vdb(image_directory, img_start_idx, None)

Processing Images: 100%|█████████████████████████████████████████████████████| 200599/200599 [9:26:04<00:00,  5.91it/s]

Quick log

from pymilvus import Collection

# Assuming 'collection_name' is your collection's name
collection = Collection(name=collection_name)

# Load the collection to refresh its state
collection.load()

# Try to fetch the number of entities
try:
    num_entities = collection.num_entities
    total_images = num_entities + len(log_error_images)    
    print(f"{total_images:,} images processed, {num_entities:,} images loaded. {len(log_error_images):,} failed to load")    
    
except Exception as e:
    print(f"Error fetching entity count: {str(e)}")

201,765 images processed, 186,954 images loaded. 14,811 failed to load

Show (some) images not loaded

def show_image_array(image_array, suptitle, celeb_id=None):
    # array_images is the list of image file paths to show
    n = len(image_array)  # Number of images
    ncols = 3  # Number of columns
    nrows = (n + ncols - 1) // ncols  # Calculate the number of rows needed
    
    fig, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(15, (15/ncols) * nrows))
    axes = axes.flatten()  # Flatten the grid to make indexing easier

    celeb = None
    for i, img_path in enumerate(image_array):
        img = mpimg.imread(img_path)
        axes[i].imshow(img)
        # axes[i].axis('off')  # Hide axes
        if celeb_id: x_label = f"Celeb ID: {celeb_id[i]}"
        else: x_label = f"{os.path.basename(img_path)}"
        # get file or celeb id    
        axes[i].set_xlabel(x_label)
    
    # Turn off axes for any unused subplots
    for j in range(i + 1, len(axes)):
        axes[j].axis('off')

    plt.suptitle(suptitle,size=21)
    plt.tight_layout()
    plt.show()

Images with error

log_image_array = []
ini=1221 #is a good index
end=ini+9

# Extract the list of images to show
for log_error_image in log_error_images[ini:end]:
    img_path = log_error_image['image_path']
    log_image_array.append(img_path)

# plot images
show_image_array(log_image_array, "Images with error. Not Loaded into Milvus")

---

Task 2: Perform Similarity Search

---

Once we have our vector embedding, we can perform image similarity search by querying our Vector DB using Cosine Distance.

For this purpose, we will do it in three use cases.

Use Case: Real-Time detection using the WebCam

In this use case I'll take a picture from my webcam to find what celebrities are similar to my face.

Step 1: take several pictures

---

# Capture Code based on Medium Post from Arthur Bauville: Display your live webcam feed in a Jupyter notebook using OpenCV
# See references at the end of the notebook

# Stop button
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Stop Streaming',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Capture Button
captureButton = widgets.ToggleButton(
    value=False,
    description='Capture',
    disabled=False,
    button_style='info', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Capture Image',
    icon='camera' # (FontAwesome names without the `fa-` prefix)
)

def view(button):
    '''
    Display Image
    '''
    cap = cv2.VideoCapture(0)
    display_handle=display(None, display_id=True)
    
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        frame = cv2.flip(frame, 1) # Flip frame horizontally

        # Convert frame for display
        _, buffer = cv2.imencode('.jpeg', frame)
        display_handle.update(Image(data=buffer.tobytes()))
        
        # Check if we need to capture this frame
        if captureButton.value:
            timestamp = datetime.now().strftime('%Y-%m-%d_%H-%M-%S')
            filename = f"{CAPTURED_IMAGE_PATH}/captured_image_{timestamp}.jpg"
            # Save Captured image
            cv2.imwrite(filename, frame)
            print(f"Image Captured and saved {filename}")
            captured_image.append(filename)
            captureButton.value = False # Reset button state

        # Check if stop button was pressed
        if stopButton.value==True:
            break
                  
    cap.release()
    display_handle.update(None)
    
    return filename 
    
# Button Events= handlers
def on_capture_clicked(b):
    # Signals that the next frame should be captured
    captureButton.value = True

# Initialize array with captured images
captured_image = []

# Display buttons and start thread
button_box = widgets.HBox([captureButton, stopButton])
display(button_box)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

HBox(children=(ToggleButton(value=False, button_style='info', description='Capture', icon='camera', tooltip='C…

None

Image Captured and saved images/captured_image_2024-05-07_02-16-24.jpg
Image Captured and saved images/captured_image_2024-05-07_02-16-29.jpg
Image Captured and saved images/captured_image_2024-05-07_02-16-33.jpg
Image Captured and saved images/captured_image_2024-05-07_02-16-43.jpg
Image Captured and saved images/captured_image_2024-05-07_02-16-47.jpg
Image Captured and saved images/captured_image_2024-05-07_02-17-03.jpg

Step 2: Get Image Vector Embedding

Let's review the images and pick one for querying the DB

show_image_array(captured_image, "My Images")

So we can pick the best one

# Number of image to do the search
image_number = 4
# Get one of the images
user_img_path = captured_image[image_number]

# Display an image file
Image(filename=user_img_path)

And extract the Image Feature (my face) from the picture using deepface

def show_all_faces(user_img, suptitle, num_columns=2):
    ext_faces = []
    num_images = len(user_img)
    cols = num_columns # Number of columns
    rows = (num_images // cols) + (num_images % cols > 0)  # Calculate the number of rows needed
    
    fig, axes = plt.subplots(rows, cols, figsize=(5, 1.5 * rows))
    
    for i, face in enumerate(user_img):
        ext_faces.append(face['face'])
        ax = axes[i // cols, i % cols]
        ax.imshow(ext_faces[-1])
        ax.set_title(f"image {i}")
        ax.axis('off')  # Hide the axes
    
    # Hide any unused subplots
    for j in range(i + 1, rows * cols):
        fig.delaxes(axes[j // cols, j % cols])

    plt.suptitle(suptitle)
    plt.tight_layout()
    plt.show()

# In case there are multiple ppl in the picture
def show_one_face(user_img, face_num):
    # inspect captured image extraction
    plt.figure(figsize=(2, 2))
    plt.imshow(user_img[face_num]['face'])
    # plt.imshow(my_img)
    plt.title(f'Face extracted # {face_num}')
    plt.tight_layout()
    plt.show()

def split_img_vector(img_vector, face_num):
    # split vector into embeddings, metadata, and confidence
    img_embedding = img_vector[face_num]['embedding']
    img_metadata = img_vector[face_num]['facial_area']
    img_confidence = img_vector[face_num]['face_confidence']

    return img_embedding, img_metadata, img_confidence

def get_similar_images(user_img_path, face_num=0):
    # Get Vector Embeddings and Metadata
    model_name = ["Facenet","VGG-Face","OpenFace","DeepFace"]
    
    # Get one of the images
    user_img = DeepFace.extract_faces(user_img_path, enforce_detection=True)
    
    user_img_vector = DeepFace.represent(user_img_path, model_name=model_name[0], enforce_detection=True)
    
    # split vector into embeddings, metadata, and confidence
    user_img_embedding, user_img_metadata, user_img_confidence = split_img_vector(user_img_vector, face_num)
    
    # Get matched objects
    matched_ids = query_collection(user_img_embedding)
    matched_images, matched_embeddings = get_matches(matched_ids)
    matched_img_path, matched_img_id = get_matched_paths(user_img_path, matched_images)

    return matched_img_path, matched_img_id, user_img, user_img_embedding

# Get image feature (my face)
user_img = DeepFace.extract_faces(user_img_path, enforce_detection=True)
# I know there is only one face in this picture, which is mine (face_num=0)
show_one_face(user_img,face_num=0)

We get the vector embedding and metadata

# Get Vector Embeddings and Metadata
model_name = ["Facenet","VGG-Face","OpenFace","DeepFace"]

user_img_vector = DeepFace.represent(user_img_path, model_name=model_name[0], enforce_detection=True)

# split vector into embeddings, metadata, and confidence
user_img_embedding, user_img_metadata, user_img_confidence = split_img_vector(user_img_vector, face_num)

Metadata and Embeddings

# Shows 10 dimensions of vector embeddings]
show_dims=10
print(f"Embedding Vector Dimension: {len(user_img_embedding)}")
print(f"First {show_dims} dimensions")
display(user_img_embedding[:show_dims])

print(f"\nMetadata {user_img_metadata}")
print(f"Feature extraction confidence: {user_img_confidence}")

Embedding Vector Dimension: 128
First 10 dimensions

[-1.5355143547058105,
 -0.21743208169937134,
 0.3877891004085541,
 -0.4349514842033386,
 0.1683000922203064,
 0.72022545337677,
 0.9845176935195923,
 1.6162538528442383,
 -0.2157813310623169,
 0.5454258322715759]

Metadata {'x': 220, 'y': 174, 'w': 153, 'h': 153, 'left_eye': (320, 233), 'right_eye': (267, 231)}
Feature extraction confidence: 0.91

Step 3: Search similar vector embeddings

Now that we have my face's vector embeddings, we can query the Vector DB to find these vectors most similars to mine

First, let's get the list of Celebrities IDs

Note: the Dataset doesn't inclue the name of the celebrities, only their numerical IDs.

# Path to your file
file_path = 'CelebA Dataset/identity_CelebA.txt'

# Read the file into a DataFrame
df = pd.read_csv(file_path, delim_whitespace=True, header=None, names=['filename', 'identity_id'])

# Helper function to get celeb Identity given a picture filename
def get_identity(filename):
    identity_id = None
    identity_ids = df[df['filename'] == filename]['identity_id'].values
    if identity_ids.size > 0:
        identity_id = identity_ids[0]
    else:
        identity_id = "Original Image"
    return identity_id

# Helper function to get the matched ids from the Vector DB
def query_collection(user_img_embedding, num_of_images = 8):
    # The embedding you want to search with
    query_embedding = user_img_embedding  
    
    # Define your search parameters
    search_params = {
        "metric_type": "L2",  # or "IP" depending on your distance calculation preference
        "params": {"nprobe": 1000}
    }
    
    # Perform the search
    results = collection.search(
        [query_embedding], "embedding", search_params, limit=num_of_images
    )
    
    # And stitch together results
    matched_ids = []
    for result in results[0]:
        matched_ids.append(result.id)

    return matched_ids

# Helper function to get the matched images and embeddings
def get_matches(matched_ids):
    matched_images, matched_embeddings = [],[]
    
    for entity_id in matched_ids:
    # Perform the query
        # Here, `output_fields` specifies which fields we want to retrieve
        query_results = collection.query(expr=f"id == {entity_id}", output_fields=["*"])
        
        # Display the results
        matched_images.append(query_results[0]['name'])
        matched_embeddings.append(query_results[0]['embedding'])
        
    return matched_images, matched_embeddings

# Helper function to get image path and image ids
def get_matched_paths(user_img_path, matched_images):

    matched_img_path = [user_img_path]
    matched_img_id = [get_identity(user_img_path)]
    
    for matched_filename in matched_images:
        matched_img_id.append(get_identity(matched_filename))
        matched_img_path.append(os.path.join(CELEBA_IMAGE_PATH,matched_filename))
        
    return matched_img_path, matched_img_id

Then we query the Collection in Milvus and get the images with smallest cosine distance

matched_ids = query_collection(user_img_embedding)
matched_images, matched_embeddings = get_matches(matched_ids)
matched_img_path, matched_img_id = get_matched_paths(user_img_path, matched_images)

Finally, we can plot the closest images

# plot images
show_image_array(matched_img_path[:6], "Matched Images", celeb_id=matched_img_id)

Step 4 : Evaluation

First Method: 2D Embeddings

To do a quick visualization that allow us to compare the embeddings, we use this trick taken from this post to convert the 1D vector embeddings into a 2D matrix embedding, so, the vectors look like a barcode that can be visualized and compared among different images.

The trick is simple: duplicate the embeddings in several rows, so we get a matrix, as shown in the function get_2D_embeddings

# Converts vector embedding into a matrix embedding
def get_2D_embeddings(embeddings, height=10):
    embedding_2d = []
    for i in range(0, height):
        embedding_2d.append(embeddings)

    return embedding_2d

# Get first elements of the list
images   = [user_img[face_num]['face']]
images = [mpimg.imread(user_img_path)]
barcodes = [get_2D_embeddings(user_img_embedding)]

# now we can iterate and add the matched images on the list 
for i, image_path in enumerate(matched_img_path[1:]):
    if i==0:
        pass
    images.append(mpimg.imread(image_path))
    barcodes.append(get_2D_embeddings(matched_embeddings[i]))

def plot_single_image_barcode(match_number):
    '''
    Plot only the match specified in the match_number
    '''    
    # The original image is always 0
    original_number=0

    # Get the images and the 2D embeddings
    img1 = images[original_number]
    img2 = images[match_number]
    img1_graph = barcodes[original_number]
    img2_graph = barcodes[match_number]

    # Now, plot logic 
    fig = plt.figure(figsize=(8, 3))
    
    # Create a GridSpec with 2 rows and 4 columns, with the second column wider
    gs = gridspec.GridSpec(2, 4, width_ratios=[1, 3, 0.1, 0.1])
    
    # Plot first image
    ax1 = fig.add_subplot(gs[0, 0])
    plt.imshow(img1)
    
    plt.title("Original Img")
    plt.axis('off')
    
    # Plot first graph, larger subplot
    ax2 = fig.add_subplot(gs[0, 1:3])
    im = plt.imshow(img1_graph, interpolation='nearest', cmap=plt.cm.ocean)
    plt.colorbar(im, cax=fig.add_subplot(gs[0, 3]))
    
    # Plot second image
    ax3 = fig.add_subplot(gs[1, 0])
    plt.imshow(img2)
    plt.title(f"Celeb ID: {matched_img_id[match_number]}")    
    plt.axis('off')
    
    # Plot second graph, larger subplot
    ax4 = fig.add_subplot(gs[1, 1:3])
    im = plt.imshow(img2_graph, interpolation='nearest', cmap=plt.cm.ocean)
    plt.colorbar(im, cax=fig.add_subplot(gs[1, 3]))
    
    plt.tight_layout()
    plt.show()

def plot_all_images_with_barcodes(images, barcodes):
    '''
    Plot all the matches in the arrays images and barcodes
    '''
    num_pairs = min(len(images), len(barcodes))
    fig = plt.figure(figsize=(7.5, 1.5 * num_pairs))
    gs = gridspec.GridSpec(num_pairs, 4, width_ratios=[1, 3, 0.1, 0.2])

    for i in range(num_pairs):
        img = images[i]
        img_graph = barcodes[i]

        # Plot image
        ax1 = fig.add_subplot(gs[i, 0])
        plt.imshow(img)
        if i == 0: img_id = 'Original IMG'
        else: img_id = f"Celeb ID: {matched_img_id[i]}"
        plt.title(img_id)
        plt.axis('off')

        # Plot graph, larger subplot
        ax2 = fig.add_subplot(gs[i, 1:3])
        im = plt.imshow(img_graph, interpolation='nearest', cmap=plt.cm.ocean)
        plt.colorbar(im, cax=fig.add_subplot(gs[i, 3]))

    plt.tight_layout()
    plt.show()

Now we are ready to compare embeddings

plot_all_images_with_barcodes(images[:4], barcodes[:4])

Use Case 2: Group Picture

What if we have a group picture? We can leverage DeepFace to extract the faces from the image, conveniently indexed so they can be individuale queried against the Vector DB

Looking for an individual face in a group

user_img_path = "images/bigbang.jpg"

# Display an image file
Image(filename=user_img_path)

We extract the faces from the group picture

# Get image feature (my face)
user_img = DeepFace.extract_faces(user_img_path, enforce_detection=True)
show_all_faces(user_img, "Image Features", num_columns=3)

Then we can pick one person from the group to search in the database

# Index for the face we want to search
face_num = 1
show_one_face(user_img, face_num)

Find vector embeddings

user_img_vector = DeepFace.represent(user_img_path, model_name=model_name[0], enforce_detection=True)
user_img_embedding, user_img_metadata, user_img_confidence = split_img_vector(user_img_vector, face_num)

Query our Database

matched_ids = query_collection(user_img_embedding)
matched_images, matched_embeddings = get_matches(matched_ids)
matched_img_path, matched_img_id = get_matched_paths(user_img_path, matched_images)

Plot similarities

# plot images
max_images = 6
show_image_array(matched_img_path[:max_images], "Matched Images", celeb_id=matched_img_id)

Or we can process the entire group at once

for i in range(0,len(user_img)):

    # Index for the face we want to search
    face_num = i
    max_images = 4
    
    # query database and plot similarities
    matched_img_path, matched_img_id, user_img, _ = get_similar_images(user_img_path, face_num)
    # show_one_face(user_img, face_num)
    show_image_array(matched_img_path[1:max_images], "Matched Images", celeb_id=matched_img_id)

Evaluating results using PCA and KMeans

We used PCA to reduce the dimensionality of the embeddings and perform a KMeans operation to observe if the images are clustered, suggesting similarity.

As observed in the plots below, the embeddings created clusters, as expected. In the 3D plot we can asses that the number of clusters are similar to the number of faces; however, not all the images are perfectly matched, as shown in the graph with faces above.

# Get the Embeddings
all_embeddings = []
for i in range(0, len(user_img)):
    face_num = i

    user_img_vector = DeepFace.represent(user_img_path, model_name=model_name[0], enforce_detection=True)
    user_img_embedding, _, _ = split_img_vector(user_img_vector, face_num)

    matched_ids = query_collection(user_img_embedding, 9)
    _, matched_embeddings = get_matches(matched_ids)
    
    all_embeddings = all_embeddings + matched_embeddings

def plot_PCA_2D(embeddings, k=6):
    # Apply PCA
    pca = PCA(n_components=2)
    X_reduced = pca.fit_transform(embeddings)
    
    # Apply KMeans
    kmeans = KMeans(n_clusters=k, random_state=42)  
    labels = kmeans.fit_predict(embeddings)
    
    # Plot
    plt.figure(figsize=(8, 6))
    plt.scatter(X_reduced[:, 0], X_reduced[:, 1], c=labels, cmap='viridis', s=50)
    plt.xlabel('PC1')
    plt.ylabel('PC2')
    plt.title('2D PCA of 128-Dimensional Embeddings with KMeans Clustering')
    plt.colorbar(label='Cluster')
    plt.show()

def plot_PCA_3D(embeddings, k=6):
    # Apply KMeans
    kmeans = KMeans(n_clusters=k, random_state=42)
    labels = kmeans.fit_predict(embeddings)
    
    # Apply PCA
    pca = PCA(n_components=3)
    X_reduced_3d = pca.fit_transform(embeddings)
    
    # Plot the 3D PCA
    fig = plt.figure(figsize=(10, 7))
    ax = fig.add_subplot(111, projection='3d')
    scatter = ax.scatter(X_reduced_3d[:, 0], X_reduced_3d[:, 1], X_reduced_3d[:, 2], 
                         c=labels, cmap='viridis', s=50, edgecolor='k')
    
    ax.set_xlabel('PC1')
    ax.set_ylabel('PC2')
    ax.set_zlabel('PC3')
    ax.set_title('3D PCA of 128-Dimensional Embeddings with KMeans Clustering')
    
    # Add color bar
    colorbar = plt.colorbar(scatter)
    colorbar.set_label('Cluster')
    
    plt.show()

embeddings = np.array(all_embeddings[:])

plot_PCA_2D(embeddings)
plot_PCA_3D(embeddings)

User Case 3: Is Cerys an undercover Celebritiy?

In our third and last use case, we will try to unveil one of the misteries of the DS community: Is Cerys Prick an undercover celebrity?

We start with the images below and then we do a search in our vector database.

# Celebrities and friends - Use case 2
captured_image=[
     "images/cerys9.jpg"    
    ,"images/cerys1.jpg"
    ,"images/emma_stone3.jpg"
]
show_image_array(captured_image[:2], "My Images")

Results after matching against our embeddings,

we get the closest match with Katy Perry in the first image, and a close 4th match in the second one.

cerys_matched_img1_path, cerys_matched_img1_id,cerys_face1, cerys_img1_embedding = get_similar_images("images/cerys1.jpg",face_num=0)
cerys_matched_img2_path, cerys_matched_img2_id,cerys_face2, cerys_img2_embedding = get_similar_images("images/cerys9.jpg",face_num=0)

# Now we can plot
show_image_array(cerys_matched_img1_path[:2], "Matched Images", celeb_id=cerys_matched_img1_id)
show_image_array(cerys_matched_img2_path[:1] + cerys_matched_img2_path[6:7], '', cerys_matched_img2_id[:1] + cerys_matched_img2_id[6:7])

Evaluating our Results using 2D Embeddings

We can use our "bar code" method to do a visual evaluation of the embeddings

# Get first elements of the list
face_num=0
images   = []
barcodes = [get_2D_embeddings(cerys_img1_embedding)]

# now we can iterate and add the matched images on the list 
for i, image_path in enumerate(cerys_matched_img1_path[:]):
    if i==0:
        pass
    images.append(mpimg.imread(image_path))
    barcodes.append(get_2D_embeddings(matched_embeddings[i]))

# Plot image
plot_single_image_barcode(1)

# Get first elements of the list
images   = []
barcodes = [get_2D_embeddings(cerys_img2_embedding)]

# now we can iterate and add the matched images on the list 
for i, image_path in enumerate(cerys_matched_img2_path[:]):
    if i==0:
        pass
    images.append(mpimg.imread(image_path))
    barcodes.append(get_2D_embeddings(matched_embeddings[i]))

# Plot image
plot_single_image_barcode(6)    

Evaluating our Results Empirically

However, empirically we know that the models' performance is not OK, because it is already stablished that Cerys is not Katy Perry, but Ema Stone, disguised as a HES student, learning her character for the role she will play in her next Data Scientist Sci-Fi Action movie.

The similitude is stablished in the graph below.

show_image_array(captured_image[1:3], "Empirical match with Ema Stone = 100% *")

$(*)$ Empirical method used: common knowledge among most students that known the Subject.

Drop Collection

# Drop the entire collection
# collection.drop()

collection.num_entities

187647

import pandas

Conclusions

Next Steps

References

• Large-scale CelebFaces Attributes (CelebA) Dataset
• Milvus Installation Documentation
• Deep Face Analysis Framework for Face Recognition and Demografy
• Display your live webcam feed in a Jupyter notebook using OpenCV
• Gentle Intruduction to face recognition in deep learning